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NASA Langley Research Center embedded four rapporteurs at the Complex Aerospace 
Systems Exchange (CASE) held in August 2013 with the objective to capture the essence of 
the conference presentations and discussions. CASE was established to provide a discussion 
forum among chief engineers, program managers, and systems engineers on challenges in 
the engineering of complex aerospace systems. The meeting consists of invited presentations 
and panels from industry, academia, and government followed by discussions among 
attendees. This report presents the major and reoccurring themes captured throughout the 
meeting and provides analysis and insights to further the CASE mission. 


I. Introduction 

T he AIAA Complex Aerospace Systems Exchange (CASE) was initiated in 2012 and held in conjunction with 
the AIAA SPACE 2012 Conference & Exposition. CASE was established to provide a forum for discussion and 
“exchange” among chief engineers, program managers, and systems engineers regarding aerospace system 
development challenges. The scope covered the entire aerospace system life cycle including design, development, 
integration, testing, modeling and simulation, and operations, with the goals of minimizing cost overruns and delays 
and mitigating system and program failures, particularly those late in the life-cycle. 

One of the issues facing the aerospace systems community is achieving comprehensive success in developing 
major complex systems. For example, in some cases, we retrospectively claim at the program or project’s conclusion 
that the stated goals were achieved and that the ultimate system design and execution was successful. In a general 
sense, success is claimed when a project exhibits technical success such as meeting the specified performance of a 
new spacecraft or aircraft implying that current engineering practices are adequate. Technical accomplishments can 
overshadow significant schedule delays, cost overruns, and many other shortcomings associated with environmental 
issues, societal impacts, and political difficulties. The bottom line is that current processes and methods may not 
ensure overall program/project success, particularly if we consider meeting the budget, schedule, and operational 
effectiveness requirements. CASE organizers have recognized that current engineering or mathematical tools are not 
necessarily deficient, however we now face an increasingly complex mix of challenges that include examining our 
internal processes, organizations, policies, contracting, and other facets that stretch beyond traditional engineering 
competencies. 

To proactively address these complex systems, AIAA created CASE and designed it to be unlike existing AIAA 
conferences such that, instead of technical paper presentations, the format was to provide a discussion within the 
aerospace systems arena emphasizing information and idea exchange where participants had the opportunity to hear 
insights, best practices, and lessons learned from recognized practitioners. Thus far, the conference has focused on 
specific tracks comprised primarily of panel discussions with the goal of providing CASE attendees with practical 
knowledge and ideas that could be directly applied to their daily work. One of the goals is that attendees develop 
broader perspectives about issues facing the engineering community. CASE places a greater emphasis on the 
effective transfer of information through a dynamic exchange of ideas and practical applications for complex 
systems. 
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The theme of CASE 2012 was to explore what was not known about Complexity Science and its application to 
engineering. During the pre-conference Executive Session, a catastrophic system failure was examined to elucidate 
key system complexities (such as training, cockpit design, etc.) and, in particular, what might have been done to 
prevent this failure in the design and implementation of the system. The 2012 conference was co-located with the 
AIAA Space Conference and included tracks on Complex Systems Development , Integration, Test, and Verification 
of Complex Systems, and Program Management of Robust and Resilient Systems. Sessions included: 

• Business Operations and Logistics 

• Collecting Our Thoughts: Complex Systems Development 

• Elegant Design and Complex Systems Development 

• Execution of Successful Programs 

• Forensic Investigations I: Problems Rooted in Design 

• Forensic Investigations II: Problems Rooted in Program Issues 

• Integration of Modeling and Simulation, Ground Test, and Flight Test 

• Lessons Learned in Integration, Test, and Verification 

• New Acquisition and Regulatory Approaches 

• New Paradigms for Complex Systems Development 

• Planning and Executing an Integration Test Strategy for a Complex Aerospace System 

• Verification and Validation Issues 

CASE 2013, held in conjunction with the AIAA Aviation Forum, continued the presentation/panel/discussion 
format but with an emphasis on successful programs and system implementations through testimonies from complex 
aerospace system practitioners. A major theme of CASE 2013 was to look at interdependencies: what is happening 
at system and discipline boundaries that may result in emergent behavior. 

A challenge was reiterated to the aerospace community to share experiences and develop theoretical rigor and 
foundational methodologies that can be broadly employed in design, development, and operation. CASE 2013 
opened with an Executive Session where invited panel members presented and discussed three successful, large- 
scale engineering projects. The subsequent days consisted of the following tracks. 

Track 1, Complex Systems Development, was concerned with meeting stakeholders’ needs for increasingly 
complex and critical aerospace systems developed within planned budgets and schedules. The focus was on the 
complete life cycle of large-scale system development activities and how to address challenges faced by chief 
engineers and systems engineers working to develop complex aerospace systems. The sessions were: 

• Lessons Learned in Complex Systems Development - Chief Engineer Perspectives 

• Making the Business Case for Model Based Engineering 

• Managing Complexity: Academic and Industry Perspective on Metrics for Complex Systems 

• Model Based Engineering Use in System Development - Case Studies 

Track 2, Integration, Test, and Verification of Complex Systems, was concerned with meeting affordable 
development goals by planning the integration of complex systems to significant detail well in advance of detailed 
design. Strategies to design and mature models, databases, simulations, and test equipment to support program needs 
extending from bench testing of prototypes through flight testing were discussed. The sessions were: 

• Development of Testable Requirements at the System-of-Systems/Capability Level 

• Cost Optimization and Risk Mitigation through Strategic Early Verification 

• Direction and Integration of Experimental Ground Test Capabilities and Computational Methods 

Track 3, Program Management to Achieve Robust and Resilient Systems, was concerned with program 
management strategies and integrated planning tools to coordinate the activities of large, distributed technical teams 
from multiple organizations so that they may work collaboratively and share common databases in the development 
and operation of complex aerospace systems. The sessions were: 

• Program Management and Project Planning 

• Supplier Management and Logistics 
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• Workforce Skills Development 

• Program Organization 

To assist in capturing the major themes from CASE 2013, the conference organizers sought out rapporteurs to 
participate in the event. The rapporteurs were tasked with distilling the key discussion points from the CASE 
sessions, including facilitated discussions with the audience. This included proposed ideas, new concepts, major 
agreements and relevant disagreements. The rapporteurs were asked to consider a wide perspective of the 
discussions, including non-traditional topics that can easily be overlooked in a traditional technical paper. Examples 
of these topics are leadership challenges, communication issues, training needed, and legal challenges. NASA 
Langley Research Center’s Complex Aerospace Systems Team (CAST) provided rapporteurs for the CASE 2013 
conference and this paper documents their findings, highlighting key challenges and opportunities that were brought 
up by conference attendees. Several attendees have generously provided their notes and insights to the rapporteurs 
and these inputs were also considered in the findings. 

The views expressed herein by the team of rapporteurs provide a summary of different perspectives on CASE 
2013. However, it is important to note that these views are not intended to be comprehensive nor fully consistent. 
The rapporteurs purposefully represent different genres of aerospace engineering and thus inherently bring diversity 
of thought to the findings provided. The paper begins with a discussion of the characteristics of complex aerospace 
systems, followed by a summary of the presentations and discussion from the conference, and concludes with a 
discussion of potential future directions. 

II. Characteristics of the Design and Development of Complex Aerospace Systems 

Overall, CASE provided a rich exchange of ideas and case studies that addressed complex aerospace systems 
and the environment in which they are developed. However, as we reviewed and compared our findings, it became 
apparent that the term “complex” is not used consistently and there were many preconceived notions brought by 
attendees. Furthermore, almost all of the participants would likely claim that they work on complex systems. 
Therefore, we recognize that there is a need to bring clarity around the topic of what constitutes a complex 
aerospace system. Defining the characteristics of complex systems is a vital starting point for advancing the 
discussion and efforts to develop new methodologies. The CASE organizers recognized this need for commonality 
in the definition and provided several read-ahead documents to prepare the participants. In this section, we 
summarize key elements from these documents and highlight agreed characteristics of the design and development 
of complex aerospace systems. 

From the executive chair’s introductory article 1 , complex aerospace systems are being developed in an 
environment with rapidly evolving technology. To leverage the latest technological components, system 
development is required to adapt, in real-time, to the introduction of disruptive technologies. Furthermore, complex 
systems exhibit emergence, the onset of unpredictable responses and consequences that are often realized late in the 
development cycle. Emergence itself may invalidate the premise of traditional systems engineering methods, since 
it cannot be adequately accommodated in the design. Lastly, Felder recognizes that today’s engineered aerospace 
systems are intertwined and embedded with socio-technical elements inseparably combining both the developers and 
the environment in which they operate. 

An additional perspective was provided by the INCOSE Complex Systems Working Group through a primer 2 on 
complexity that was made available in draft form to CASE participants prior to the conference. Overall, their primer 
provides an excellent overview and is particularly well suited for practicing systems engineers. Complexity spans 
many domains, well beyond traditional engineering disciplines. The Working Group emphasizes that complexity 
refers not only to the systems themselves, but also to the environment in which the systems are designed and 
developed. Environments that include many technical, organizational, economic, and societal stakeholders, with 
competing objectives that are often concurrently evolving as the system is being developed. The INCOSE working 
group noted the following attributes of complex systems. 

• Are inherently uncertain and unpredictable, and we cannot fully understand its structure or behavior. 

• Cannot be adequately reduced into parts, recursively designed and re-assembled to form the whole solution. 

• Contains autonomous parts whose interactions lead to emergent self-organized patterns of behavior. 

• Created by the interaction of people, organizations, and the environment. 

• Results from the diversity, connectivity, interactivity, and adaptivity of a system and its environment. 

• Unintended consequences can overwhelm or negate the system’s design. 

• Cannot be described at a single level; multi-scale descriptions are needed to understand complex systems. 
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• Emergent behavior, derived from the relationships among their elements and from multiple internal feedback 

loops, gives rise to observed patterns that may not be understood or predicted. 

• Often impossible to predict all future configurations, structures, or behaviors of a complex system, given 

finite resources. 

• Perpetually generating novelty, many key variables are opaque, boundaries are indeterminate, and weak ties 

can have a disproportional effect on system behavior. 

For our final perspective, we consider the views of the Interagency Working Group on Engineering of Complex 
Systems established to promote the rethinking of engineering methodologies for large complex systems critical to 
our national security, economy and quality of life. These systems must be resilient in the face of natural disasters, 
creative adversaries, and an unforeseeable future. Collectively 3 , the working group has determined that a system 
may be considered complex if it is not mathematically predictable within reasonable constraints of time, 
computational power, and existing modeling tools. The interactions that can take place within a complex system 
cannot be fully imagined or defined based on an understanding of its constituent elements. This makes it impossible 
to fully test such a system before deployment in a traditional manner. Beyond the unpredictable behaviors of these 
systems, the inability to fully understand and envision them during their design phase usually leads to significant 
cost and schedule overruns in the development and operation phases. 

At the CASE conference, several speakers provided examples of the above characteristics. We reference a few 
salient comments from the conference below. 

• Many systems have second- or third-order phenomena that can cause first-order responses. 

• Many failures are the combination of a few little things going wrong, a synergistic effect of weak signals. 

• A certain amount of irreducible uncertainty exists and at some point you must accept it and even embrace it. 

• While we aspire to create clean interface lines between system sub-elements where there are as few variables 

as possible passing between sub-element boundaries to reduce coupling, the system interfaces are often not 

determined by engineering strategy but by historic technical disciplines or organization structure. 

• Determining the necessary human aspects of systems development (training, communication strategy, 

stakeholders, co-location, and politics) can be just as complex as determining the technical aspects. 

In summary, the general characteristics of complex systems include interdependencies, emergent behavior, 
adaptive and resilient requirements, and socio-technical elements. Consistently, these characteristics of complex 
systems include elements that are often unpredictable using conventional modeling and abstraction methods of 
system design, development, testing, and operation. Recognizing the attributes of complexity is the first step to 
motivate the development of engineering methodologies to address these challenges. 

III. Setting the Conference Tone and Objectives through Case Study Exploration 

The CASE 2013 conference was preceded by a one-day Executive Session of invited attendees. Some important 
themes were introduced setting a tone that would be reiterated throughout the conference. Participants generally 
agreed that research into improving the engineering of complex systems is required to progress beyond a state 
dominated by heuristics based on success stories to a state grounded in sound theories and proven methods. 
Through discussion, it was recognized that the community should document processes and methods leading to 
strategic and consistent success in the establishment of theorems and methodologies that can be scaled, taught, and 
applied to complex aerospace systems. Regarding today’s state of practice, it was noted that considerable 
documentation exists on methods, processes and tools (MPTs) for engineering large systems and that these MPTs 
have proven successful. Yet, sufficient examples also exist to demonstrate that we are currently designing and 
building systems that do not behave as intended despite following current protocols. Furthermore, the dynamics and 
complexity of the challenges that are faced today are such that the engineering community is not able to rely solely 
upon existing protocols to guide the engineering of complex systems for the future. Rather, a strategic goal may to 
be to develop underlying theories, principles, and defined practices that are teachable, scalable, and repeatable. 

Attendees suggested several opportunities and challenges toward reaching these goals, three of which were often 
repeated. First, discussions on engineering complex systems have a tendency to be confined to a discussion of 
System Engineering (SE) with inadequate consideration of other relevant disciplines, such as optimization, decision 
science, and organization science. SE practice and protocols were noted to be an essential but insufficient ingredient 
in engineering complex systems. Second, the applicability and relevance of research into complexity is not clear. 
While complexity is being studied and applied in many genres and large aerospace systems share some 
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characteristics with systems that are being studied in complexity research, a well-defined approach to applying 
complexity research to the engineering design, integration, and development of aerospace systems has not been 
defined. Third, in describing the current research in engineering complex aerospace systems an analogy was made 
with building construction. However, the community appears to be erecting many buildings that are on different 
foundations such as complexity science, systems engineering, decision analysis, systems analysis and optimization, 
and organization science, which are not completely consistent. A criticism was made that in some instances papers 
in engineering complex systems are being written citing only those that agree with its findings while giving 
insufficient regard to those that do not. Achieving theoretical rigor and consistency in the research on engineering of 
complex aerospace systems was noted as an important step for the future. 

Many of the overarching themes, are summarized by the phrase tyranny of the narrative, which was use 
repeatedly through the conference. Basically, it means that in retrospect we may erroneously evaluate the system 
design and development methodologies based on the success or failure of the mission. For example, even if we use 
best practices, the system could still fail. Conversely, some projects have succeeded despite poor organizational 
practices. In these cases, mission success does not necessarily indicate that the organization is performing well. The 
warning is that this common practice of retrospectively reshaping our narrative does not advance rigorous research 
to objectively evaluate the efficacy of system design and development methodologies. Therefore, we need to 
develop consistent, sound theories and proven methods to build our approaches. As the community moves forward, 
this concern is especially relevant since many future missions do not have adequately similar predecessors. 

Following the opening session, three highly successful engineering programs were presented and discussed. 
While all three programs met their technical requirements for success, each faced tremendous challenges and noted 
that failure was always eminent. Further, many non-engineering facets were noted that were critical to their 
successes such as team formation, communication, and stakeholder buy-in. Throughout the executive session and 
panel discussion, four primary themes emerged: team dynamics, system design / architecture, modeling and 
simulation, and socio-technical aspects of complex systems. 

Concerning, team dynamics, it was agreed that managing a team requires a constructive tension among members 
on the team and a need to foster a culture that leverages this constructive tension. Improved team culture enables 
people to make the right decision via increased sharing of actionable knowledge and effective dissemination to the 
team. Such a culture enables teams to aggressively challenge technical positions without the perception of a personal 
attack. Development of this culture and management of engineering teams in general was often noted as a critical 
element in system development. It is important that any system development possess a clear, accessible, and 
pervasive goal that translates to every team member’s level. 

Team composition and dynamics were often noted as important facets of program success. Two positive 
characteristics of teaming that were highlighted included: working with long standing employees who have worked 
together for a number of years and had a healthy personal rapport and keeping meetings candid and open to provide 
a healthy intellectual tension. In essence, the team dynamics could be personally respectful, while challenging each 
other intellectually. Though an overarching goal was to instill a holistic, systems level understanding in every team 
member, the project’s chief engineer stated, “You don’t make system engineers; you find them.” Team member 
experience was vitally important and enabled the application of proven methods from previous missions. 

The discussion highlighted that today’s aerospace systems involve development teams and companies that are 
distributed globally. This promotes collaboration between multiple companies to take advantage of “best in class” 
contributions. Modular architectures were employed allowing parallel path development to reduce life cycle costs. 
Technical decisions were delayed until readiness was proven. 

Concerning the system design / architecture, interfaces must be established such that interactions are minimized 
with less chance for emergent behavior. This, in turn, may offer potential for leverage to tease out new interactions 
to increase system performance. Engineers are encouraged to embrace the idea of irreducible uncertainties and 
accept or leverage them to drive toward system solutions that are relatively insensitive to uncertainties. Complex 
systems have a tremendous increase in operational states that often cannot be fully explored. It is not the risks that 
have been analyzed but the un-analyzed risks that will likely lead to failures. Also, computing or managing risk for 
very rare events is a significant challenge. Differing from a traditional approach, the question was posed as to 
whether the first iteration must succeed or can we accept rapid prototyping failures. 

In general, most of the methodologies presented tended toward a sequential, reductionist approach. Although the 
three case studies met their mission successfully, it was noted that it is not always possible to mitigate uncertainty to 
the point of acceptable risk. It may not be possible to know enough about a complex system and all of the 
interactions of its components to guard against undesired emergent behavior. One speaker suggested to “aspire to 
identify what you will never completely know” about the system. Reducing analysis to simpler components may 
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reduce risk for that component, however it does not guarantee that, when this component interacts with others, it will 
behave as predicted. 

Understanding changing societal requirements and properly responding to those requests in the design and 
implementation during system development was identified as a challenge. For example, users could immediately see 
potential alternative uses of systems, beyond their initial capabilities for alternative applications. In some of these 
extended applications, societal implications such as privacy and safety may become limiting factors rather than the 
technical system viability. Furthermore, environmental and societal changes can result in substantial changes in 
system operational needs. Many of these changes are often unpredictable, such as the future use and operational 
scenario. Significant, unpredictable changes to system requirements force the system to be used or redesigned for 
missions not originally planned resulting in a ripple affect in terms of cost, business strategy, safety, and risk. This 
consideration is a departure from the “textbook” teaching of good program management and systems engineering 
where the focus tends toward, and the process often depends upon, freezing requirements. 

Concerning modeling and simulation, system performance and margin dynamics have been proven to be 
tractable using rapid modeling and simulation to ensure the system meets objectives and ensures customer 
confidence in corrective decisions. Traditionally, the systems approach is to design for overall performance. Now, 
modeling and simulation allows trades and dynamic reallocation among subsystem components to improve overall 
performance. Model fidelity must capture not only what system the team thinks they are designing, but also 
recognize that emergent complex system behavior may occur that cannot be modeled. 

Among the noted challenges was significant uncertainty in system operational conditions. In addition, in some 
cases the system could not be fully tested prior to mission execution, resulting in the mission serving as the first 
complete system test. One strategy was to design the system such that any behavior resulting from uncertainties 
would not cause failure. Essentially, design the system to be robust to uncertainty thereby reducing the impacts of 
potential emergent behavior. To mitigate risk, they opted for small simulations of high confidence that were 
interrelated instead of a large simulation of low confidence. With every step of design, choices in assumptions were 
made, understanding that error is additive. 

Several case studies included model based engineering (MBE), also known as Model Based Systems 
Engineering, approaches to complex aerospace systems studies. In aggregate, great momentum and progress was 
shared in the successful developments across these varied applications. Common benefits to complex systems design 
and analysis included early and meaningful discussion across disciplines with stakeholders. In doing so, these 
examples benefited in the end by capturing essential cross discipline linkages and/or unrecognized stakeholder 
couplings within the system boundary that required specific action to address. Second, all the examples illuminated 
the benefits of documenting requirements and maintaining their traceability. This had the positive impact of 
reducing imprecise communications across the various activities and disciplines in the complex systems 
development or study team. An important benefit of MBE, is that enabled the construction of a wide and complex 
study of alternatives that would have been intractable without stakeholder agreement. Furthermore, the study would 
have been challenged with where to even start and how to cover most scenarios. Academic research shared in this 
session points to the potential for formal methods to substantiate the MBE application to complex systems by 
informing the designer of all the possible input scenarios in a conceptual complex system and inform the sufficiency 
of the MBE approach to clearly satisfy the design objectives. Challenges to successful MBE implementation 
included a discussion of a lack of standards, portability challenges in a global environment, and high costs to effect 
calibration and configuration control. 

Concerning socio-teclmical aspects of complex systems, collaborations throughout the life-cycle must also 
include political, economic, safety and overall system value with broad stakeholder community. Views of risk by a 
company may be very different from stakeholders, users, and regulators. Engineers should not try to treat all systems 
as physics/engineering problems, but remain curious as to other fundamental drivers. For better designs of complex 
systems, the workforce needs education, mechanisms to develop experience, improved processes, along with 
advanced modeling and simulation. 

Overall, the opening day of CASE set the tone of the conference and identified many of the particular challenges 
in designing today and tomorrow’s aerospace systems. Throughout the discussion of case studies, the projects were 
faced with many challenges that are outside of traditional systems analysis, engineering, and project management 
bodies of knowledge. Moreover, following existing design and management protocols was a necessary element to 
project success, but it was evident that these protocols were insufficient to improve the likelihood of success. Many 
of those challenges were reiterated in subsequent sessions discussed in this paper. During the opening session, there 
was an excellent exchange of ideas and experiences and an intense interest combined with a sense of urgency to 
develop methodologies to address these challenges. 
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IV. Leadership in Pursuit of Systems Thinking for Individuals and 
Organizations to Achieve Robust and Resilient Systems 

A topic area was devoted to leadership and management challenges within large complex projects and programs 
highlighting distinct aspects of concurrent engineering and the associated risks and difficulties encountered when 
inserting new technology. Advancing technology pushes designs toward the complex due to the dominant trends of 
increasing capability and the expanding role of automation. The resulting system decomposition and interface 
definition further encourages simpler, better quantified boundaries driving an individual’s knowledge of 
requirements toward multidisciplinary levels of system thinking. Instilling the aspect of all-knowing-system- 
understanding at an individual and group level requires a conscious effort to fundamentally communicate and ensure 
understanding. The inability of any one person to comprehend the entire system leads to the need for a distributed 
understanding that demands greater ability to effectively communicate that understanding. Program/project leaders 
must strive to instill system thinking at all levels of their organization and throughout the lifecycle to achieve robust 
and resilient systems in light of ever increasing complexity. 

Although many see planning as central to program management during the early part of the lifecycle, it is also 
important to remember that the planning effort is continuous throughout the lifecycle. Planning should be “living” 
and placed in a mode of constant update and adjustment to stay in tune with the challenges and issues as they arise. 
These constant updates require the entire team to ride the crest of the wave, peering over the edge and recognizing 
emerging issues before they actual become issues. Effective program leaders employ communication constructs and 
actively prevent isolating silos that limit awareness and communal action. 

Integrated master plans and schedules are tools that capture and communicate the path forward. Management 
should be open to alternate paths in light of encountered complexity. Meanwhile, practitioners responsible for these 
integrated products are often times constrained by budget/schedule. When issues are encountered, practitioners 
continue to push forward to resolve the issue. Issue resolution possibilities should include opportunities that might 
change direction. There are usually early indicators of an impending issue. Good leaders in tune with their team will 
encourage shared recognition by the team and allow them to adjust to the right path. Dealing with recognized 
emerging issues is considered an opportunity and exploited. Obviously, to take advantage of an opportunity requires 
change; hence the importance of nimbly controlled update is important. Integrated master plans exist to capture 
change and communicate new direction. 

The discussions emphasized that listening is critical to distinguish and recognize emerging issue/opportunity. 
Effective managers must be in tune with their team to be responsive, and not just focused on execution of the current 
detailed plan. The flip to “listen mode” is required and is a differentiating characteristic of “leadership” versus 
“management”. Placing a team in a “paused reflection mode” enables listening and enables recognition of current 
state and emerging issues. Periodic planned pause points are often used where “pencils down” is declared allowing 
time to review the product internally and with the customer. Key milestones are defined as part of the basic 
schedule plan that includes such pause points. This feedback strategy contains a framework that describes the 
recognized state. 

The Cynefin framework provides an organizational feedback strategy that utilizes a definition of four problem 
domains, the boundaries between them, and a central “Disorder” domain (the anarchy that results when individuals 
revert to self-decisions). The Cynefin 4 framework provides a context to describe the current state and enables the 
team, as a whole, to knowledgeably address the current state, and simultaneously be aware of pending state 
transition boundaries. It provides situational awareness and a framework for management. The four problem 
domains are: 

• Simple, where cause and effect is obvious to all with one “right answer”, 

the approach is to Sense - Categorize - Respond and we can apply best practice. 

• Complicated, where cause and effect requires analysis and/or expert knowledge, 
the approach is to Sense - Analyze - Respond and we can apply good practice. 

• Complex, where cause and effect only clear in retrospect, 

the approach is to Probe - Sense - Respond and we can sense emergent practice. 

• Chaotic, where there is no relation between cause and effect at the systems level, 
the approach is to Act - Sense - Respond and we can discover novel practice. 

Finding leaders who can grasp changes in the state/domain is difficult. These types of leaders usually possess a 
broad base of experiences that includes strong mentoring relationships. Real-world experience ensures grounding to 
lessons-learned and prevents recurrence of issues. Building on the “pause” theme, good leaders also grasp the need 
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for real-time “pause and learn” events to infuse risk/issue/opportunity experiences within the team. Such events also 
create mentorship opportunities between team members leading to the idea of continually feeding forward. Skillsets 
must be attuned to change and the need for the greater good. 

Organizational constructs should be in tune with the state of development to permit active, continual response. 
Functional organizations are focused on the people and their performance as opposed to a product based 
organization whose focus is primarily cost and schedule. Care and feeding for the workforce has many challenges: 
knowledge management and transition, need for instant learning, use of emerging technology, multiple generations 
in the workplace. Coaches/mentors can be leveraged to help work through the challenges. Workforce expectations 
are changing with new young hires leaning toward the faster track (e.g. hired on Monday and become CEO on 
Friday). Typically, the younger workforce embraces job transition and does not seek jobs that provide full career 
stability. Knowledge transfer has also changed to a view of instant need. Connection through social media provides 
instant contact enabling quick query response. Online information enables instant information that can make 
traditional education methods (i.e. classroom and teacher) obsolete. For example, internet videos provide quick 
“how to” information in a very efficient, low effort format. Embracing such changes often puts young and old 
generational groups at odds from a speed perspective. The quandary is the translation of life experiences between 
the generations. Mentoring roles become more important to foster and enable information transfer in an acceptable 
manner. The pull and push in a mentoring situation is very much driven by the new environment in today’s fast 
paced information world. The faster environment will have a tendency to drive the mentoring exchange harder to a 
pull scenario, so there must be awareness not to leave good information on the table. 

After exploring these characteristics and challenges of system development, we ask, “how can the untapped 
potential that exists in people be used to do better with complex aerospace systems?” System engineering can be 
viewed as a triangle comprised of process drivers, management, and engineering. These three corners combine to 
generate a holistic view brought about by system thinking. The fundamental job of the system engineer is to get the 
project team to self-realize how they participate and influence the system perspective. The question “What is the 
best?” means to instill system thinking. Is system thinking enabled through training or through experience? 
Exposing people to short-life cycle programs allows them to gain experience and to learn from the consequences of 
their decisions and behaviors. Rapid turnaround programs also allow incorporation of lessons-learned which 
enables interest and prevents burnout. The question of “are good system thinkers found or made?” revolves in the 
arena where those “found” have proven track record (i.e. experience), but more often than not, they also have 
fundamental training (i.e. are “made”). There is also the thought of enabling all engineers to be system thinkers with 
“system” emphasis promoted to the forefront prior to specializing in a focused discipline. This preparatory idea 
follows the academic model of using the undergraduate learning experience to define a level of awareness that 
prepares for the higher learning needs of graduate work. 

One view is that system thinking is the ability to see the first derivative of everything. It is seeing cause and 
effect. Every engineer’s work interfaces with others, so the need to understand global system effects are important 
and should be instilled and formalized early. Through understanding of many interfacing players, comes the need 
for humility and the notion that discipline expertise must sometimes yield the optimum for the greater good of the 
system. A level of humility acknowledges that there are other possibilities in solving a problem. Humility also 
evolves from an innate need to continually learn, forming the fundamental basis of “experience”. Again this 
emphasizes the need for engineering curriculums to include a fundamental “systems thinking” experience as part of 
the introductory engineering coursework. 

Should the inclusion circle of system thinking go beyond the technical? The business perspective very much 
plays against the technical but many times is organizationally divided from the technical. Engineering 
undergraduate programs usually do not include business classes. There is need for academia and business to foster 
well-rounded professionals expanding the purview of system thinking application. 

Exposure to real world project-based examples in college is another method of expanding experience and should 
include technical and programmatic aspects. As children, we typically learn how to consider the system more as a 
whole. Somehow this aspect gets drummed out in secondary education. Introducing this idea back into the college 
environment will better prepare students for real world problems. 

Typical discriminators used in evaluating those with higher education are not only the coursework, but also the 
outside experiences such as hobbies, projects, and co-op/intern experiences. Such realization indicates a value of 
broad experience base. Many times professors are only rewarded for academic experience, but can we find means to 
reward those that also have industry experience? 

The fundamental pursuit of system thinking is necessary to instill in this era of ever increasing complexity. 
Leadership must recognize that system thinking extends to both the individual and organizational levels. 
Maintaining “tune” to the system and focusing on communication and understandings of system behaviors is 
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critical. Although responsibility begins with the individual, it is key that norms be pushed and maintained at the 
organizational level. Leadership is key to bringing this to fruition. 

V. Test and Evaluation of Complex Systems 

Recognizing the challenges in test and evaluation for complex aerospace systems, the CASE provided a forum to 
exchange ideas on this topic. Discussions spanned from large-scale, distributed systems-of-systems to discipline 
specific computational physics, highlighting the challenges throughout the technology development, maturation, and 
operational stages. In these sessions, it was recognized that complex aerospace systems require an iterative, learning 
approach of exploration throughout the solution space and project life-cycle. This approach recognizes that there are 
elements of system performance that may not be adequately predicted in advance, often referred to as emergent 
behavior. Emergent behavior often comes from unforeseen interdependencies among system elements or the 
insufficient models of known interactions. In addition, emergence can result from unintended applications or usage 
of systems, particularly involving the coupling of human behavior with system performance. The participants of 
these sessions recognized that current design and development methodology relying on a linear, sequential process 
of development with clearly articulated performance requirements is likely not sufficient for the efficient and 
effective design of complex aerospace systems. Throughout, discussions explored the characteristics of these 
systems and motivation to adapt and extend current methodology. 

The session entitled Development of Testable Requirements at the Systems-of-Systems (SoS)/Capability Level 
studied large-scale, distributed networks of systems for national defense applications. Systems of this nature are 
developed to expand current capabilities for mission support. It was emphasized that they are intended to produce 
an effect, for example cause the opponent to capitulate, which is differentiated from satisfying the requirements. In 
fact, the session proposed that success should be based on whether the desired effect is achieved in operational 
effectiveness, not whether the requirements are met. It was discussed that while requirements are a one-to-one 
mapping to features and characteristics of a system, capabilities are not a one-to-one mapping into specific testable 
characteristics. 

SoS development occurs within a broad context including multiple stakeholders and technological disciplines, 
geographically distributed teams, and over relatively long time periods. While the attendees of the session tried to 
restrict their conversation to traditional engineering elements of test and evaluation, a recurring theme was that SoS 
performance is highly dependent on other factors, including political, organizational structure, legal, and 
procurement policies. These other factors are implicitly recognized from experience, but often are not strategically 
addressed in the development of systems with existing methodology. 

As the term connotes, SoS performance requires the integration of multiple systems, many of which are 
sufficiently complex themselves, to provide a new capability or enhance existing performance. Elements of existing 
heritage systems are often integrated with new technologies in ways that the original design requirements never 
envisioned. In the development of new system elements, the idea of “corporate citizenship” was proposed as a 
necessary aspect, meaning that each system developer has a responsibility to consider their interaction with other 
systems components. This promotes the idea of system thinking throughout all team members in the development 
process rather than relying on systems engineers to solely perform that function. 

Beyond the challenges of the technological elements, SoS performance is dependent on human operators and 
their interactions with the technology. Human interactions are a major driver of complexity since we can be quite 
unpredictable. Furthermore, it is difficult to fully simulate the actual operational environment and associated 
stressors on the human operators during the test and evaluation process. Lastly, in national defense applications, it is 
particularly difficult to predict the response of the opposing force since they may not be well-trained or well- 
behaved. Combining these elements of technology and human-centered ambiguity and performance requires new 
perspectives for testing of complex systems. 

In summary, it was proposed that test and evaluation in the SoS context should be thought of more appropriately 
as evaluation and test. In the Cynefin framework discussed previously, this is a probe and sense approach. It should 
be a continuous learning environment where we expect that systems will not work the first time, providing an 
opportunity to learn and mature the system. These are not failures, they provide vital information to improve system 
performance that cannot be fully predicted from modeling and simulation since many interdependencies are 
discovered rather than anticipated. This approach was also referred to as characterization rather than testing. The 
goal of characterization is to learn about a system’s behavior. In contrast to testing that is primarily intended to 
confirm intended performance. Characterization involves a series of experiments to understand SoS characteristics 
and to obtain feedback. 
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Discussions throughout the sessions provided compelling information that current design, development, and 
testing methodologies may not be well-suited for large-scale, distributed systems-of-systems. An opportunity exists 
for research and development of methodologies that incorporate these significantly different perspectives on both 
defining system success metrics and evaluating them. 

VI. Discussion: The Future May Not Be Where We Are Looking Today 

In this section, we provide discussion and analysis of the CASE themes and suggest some paths forward to 
develop methodologies that address the challenges of complex systems. Retrospectively, we acknowledge that the 
development of aerospace vehicles and systems over the last century is one of the greatest success stories in human 
history. However, as we have highlighted characteristics of complex systems, we recognize that many problems 
have arisen that may not be solvable with current methods. The costs in money and time of design, testing, and 
delivering new systems is accelerating at an unsustainable rate, the number of highly technically trained people and 
organizations required is dramatically increasing and managing such workforces is challenging, and systems often 
do not perform as they were intended. 

Maintenance and operations are relatively new and growing aspects of the design that is not unique aerospace; it 
is also experienced in the automotive, computer, and other industries. In space exploration, the challenges of 
operating over very long distances with long communication delays requiring increased autonomy and/or 
independence of Earth-based assets. Future space systems are focused on long duration, beyond low-earth-orbit 
operations based on scalable architectures requiring decades of investment to design, develop, and implement. 

Systems may initially perform as designed and may do so for some time but when they break, it is often difficult 
or impractical to correct problems. The complexity of designs increases the probability of intermittent problems, 
making it difficult to repair what is not broken at the time of inspection. Much of the added complexity is 
exacerbated by the integration of information technology into mechanical system, spawning an area of research 
known as cyber-physical systems. 

Perhaps refining or otherwise improving existing methods and processes can solve these problems. But other 
problems in aerospace go beyond designing and building airplanes and spacecraft. For example, the Air Traffic 
Management (ATM) system controlling the National Air Space (NAS) is rapidly reaching a saturation point. The 
hub and spoke architecture, based on centralized ground control under human supervision, is not scalable to meet 
future demands. General Aviation (GA) traffic must interface and cooperate in the current ATM. In addition to this 
unsustainable system design, the growth in commercial and GA traffic and autonomous systems will add 
substantially to the traffic load, particularly as Unmanned Arial Systems (UAS) move from remote piloted vehicles 
to totally autonomous robotic vehicles operating in the NAS. This will not only facilitate robotic services like 
package delivery but will also enable On-Demand Mobility (ODM) with Personal Air Vehicles (PAV). 

Autonomous systems will not only facilitate autonomous robotic vehicles but will develop an important role on 
piloted vehicles. Automated systems enabled the reduction from three manned crews to two on commercial vehicles 
and future improvements in autonomous systems may further the reduction to one pilot and greatly reduce the 
number of controllers required to operate space missions. Autonomous capabilities will also improve systems to 
manage vehicle health. Such autonomy in machines requires a human level of cognitive capability. Pilots are highly 
trained but they also benefit from life experience and are thus able to respond and react to unforeseen circumstances 
in creative, unscripted responses. Autonomous machines must do more than execute predetermined scripts, they 
must make decisions and adapt to new situations. 

Thus, addressing future challenges in aerospace is not simply to do what we know how to do now better: we 
need to do things we currently do not know how to do. So, how do we leant to do things better, if that is sufficient, 
or else, how do we learn to do what we do not know how to do and how do we tell the difference? Perhaps the 
barrier obstructing our progress to these solutions is found in the nature of engineers and in their training. When 
confronted with a problem, an engineer will look in his toolbox for the right tool to apply. But what if no tool in the 
box can correct the problem? The engineer will attempt to modify an existing tool to satisfy the need. A theme 
reiterated in many forms, yet converging on the same concept, appeared throughout the conference: 

• It is not the things you are doing that result in failure; it is the things you are not doing. 

• It is not the things you are talking about that result in failure; it is the things about which you are not talking. 

• It is not the things you see that result in failure; it is the things you do not see. 

• It is not the risk you are examining that result in failure; it is the risks that you are not examining. 
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In essence, it may require a retraining of the intuitive engineering approach to stop looking in the toolbox for a 
tool that is not there and start thinking of new tools. When reviewing either a success or a failure, humans tend to 
build a story around facts of the event but the story is based on things that we know; not on the unknowns that may 
have affected the outcome. A keynote speaker at CASE 2013 calls this the narrative fallacy 5 , which refers to a 
powerful, innate belief that, if we examine a problem close enough and in enough detail, we can understand 
anything in nature. But as we build our story around the facts, we limit reality to what we know and ignore the 
unknowns and, furthermore, the facts often are morphed to fit our story. Whenever we attempt to build a model of 
reality, we are reminded that "essentially, all models are wrong, but some are useful.” 6 Furthermore, our models are 
subsets of reality. Trust in a model therefore subjects one to the risks from reality outside of the subset upon which 
the model is based. 

In our attempts to build more resilient systems, the approach is to sufficiently expand our subset, however even 
the most resilient systems will still face some unconsidered circumstance. It seems implausible to expand the subset 
of what we have considered to guarantee proper behavior when confronted with reality. Biological systems like a 
muscle can actually become stronger when confronted with adversity, in contrast to engineered systems that often 
fail when stressed. 

To successfully solve some of the problems facing engineering, we have to overcome our presuppositions on 
solution approaches by looking beyond what we know and consider what is unknown and accepting what is 
unknowable. We should recognize that it might not be possible in today’s aerospace systems to examine the system 
in sufficient detail, assuming all things are predictable. 

We need to learn to build systems that do more than survive adversity; rather the system should actually learn to 
improve its response and become stronger to meet future circumstances. It seems unlikely that solely evolving and 
improving current methodologies can meet these objectives. New methodologies are essential, backed by theory, 
proven principles, and defined practice. Specifically, engineering must discover how to deal with the effect of 
interdependencies among components of large complex systems. The potential for emergent behavior and the affect 
of uncertainties and unknowables must be embraced, understood, and included in design. It is not likely that better 
designs and higher fidelity models will address these challenging aspects. 

VII. Conclusion 

In conclusion, we witnessed significant progress in advancing a common understanding of the challenges in the 
design and develop of complex aerospace systems. Coalescing common descriptors of complex systems is a vital 
starting point for advancing the development of engineering methodologies, leadership preparation and experience, 
and perspectives on system evaluation. It was emphasized that the community should strive to document processes 
and methods leading to strategic and consistent success. However, for complex systems the engineering community 
may not be able to rely solely upon existing protocols. Rather, a strategic goal is to develop underlying consistent, 
sound theories and identify proven methods that are teachable, scalable, and repeatable. Overall, to achieve our 
most challenging mission objectives it will likely require that we cannot simply do what we know how to do now 
better, rather we need to do things we currently do not know how to do. 

Furthermore, the fundamental pursuit of system thinking is necessary to instill in an era of ever increasing 
complexity. The inability of any one person to comprehend the entire system leads to the need for a distributed 
understanding that demands greater ability to effectively communicate. Leaders must strive to instill system 
thinking at all levels of their organization and throughout the life-cycle to achieve robust and resilient systems in 
light of ever increasing complexity. Effective leadership is key to bringing this to fruition. 

Lastly, we recognize that evaluating complex aerospace systems is an iterative, learning process of exploration 
throughout the solution space and across the project life-cycle. This approach recognizes that there are elements of 
system performance that may not be adequately predicted in advance, often referred to as emergent behavior. A 
learning environment recognizes controlled experiments as vital information to improve system performance that 
cannot be fully predicted from modeling and simulation since many interdependencies are discovered rather than 
anticipated. 

CASE provides a unique forum to discuss and exchange these ideas surrounding complex aerospace systems. 
The conference represents a groundbreaking effort to acknowledge that traditional approaches may not be sufficient 
to address our future needs in the aerospace community. There exists significant passion and investment by industry, 
academia, and government funded research in developing new methodologies that build on the foundations of 
current engineering analysis and design and reach beyond traditional engineering disciplines to meet the multi- 
faceted challenges of complex systems. Achieving this objective requires broadening the disciplines and expertise 
included in the exchange of ideas, many of which have not been considered part of the engineering design process. 
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Furthermore, coordination across professional societies outside of AIAA that have similar initiatives to address 
complexity is essential to identify overlap of efforts and find opportunity spaces to complement each other. Overall, 
CASE 2013 was successful in achieving its objectives and furthering the vital conversation on complex aerospace 
systems and building a community of interest that motivates research and practice. 
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